In the latter half of the twentieth century, there began a phenomenon known as the information revolution. While the information revolution is a historical development broader in scope than any one event or machine, no single device has come to represent the information revolution more than the digital electronic computer. The development of computer systems has surely been a revolution. Each year, computer systems grow faster, store more data, and provide more applications to their users.
Modern computer systems may be used to support a variety of applications, but one common use is the maintenance of large relational databases, from which information may be obtained. A large relational database is often accessible to multiple users via a network, any one of whom may query the database for information and/or update data in the database.
Database systems are typically configured to separate the process of storing data from accessing, manipulating, or using data stored in a database. More specifically, database systems use a model in which data is first stored and indexed in a memory before subsequent querying and analysis. In general, database systems may not be well suited for performing real-time processing and analyzing streaming data. In particular, database systems may be unable to store, index, and analyze large amounts of streaming data efficiently or in real time.
Stream-based computing, also called data streaming, has been used to more effectively handle large volumes of incoming data in real time. In a data streaming application, data moves through a connected network of “processing elements” called a “graph”, each processing element performing some function or functions with respect to the data.
Many data streaming applications produce data which is ultimately integrated with a relational database, i.e., it becomes new or updated entries in one or more tables in a relational database. Due to the capability of a data streaming application to handle large volumes of data, there could be a large number of new or updated entries in a database table, so large that it could be significant in comparison to data already in the table.
Conventional database query and other management techniques take no account of data which is likely to change in the near future, and in particular, of in-flight data in a streaming application which may affect the future character of the database. This may cause query results to be obsolete or misleading, or may cause certain management decisions or actions to be based on stale information.
A need exists, not necessarily recognized, for improved database management techniques which take account of future database changes, and in particular, which take account of in-flight data in a data streaming application that will or is likely to become part of the database.